Towards Understanding the Nonverbal Signatures of Engagement in Super Mario Bros

In this paper, we present an approach for predicting users’ level of engagement from nonverbal cues within a game environment. We use a data corpus collected from 28 participants (152 minutes of video recording) playing the popular platform game Super Mario Bros. The richness of the corpus allows extraction of several visual and facial expression features that were utilised as indicators of players’ affects as captured by players’ self-reports. Neuroevolution preference learning is used to construct accurate models of player experience that approximate the relationship between extracted features and reported engagement. The method is supported by a feature selection technique for choosing the relevant subset of features. Different setup settings were implemented to analyse the impact of the type of the features and the position of the extraction window on the modelling accuracy. The results obtained show that highly accurate models can be constructed (with accuracies up to 96.82%) and that players’ nonverbal behaviour towards the end of the game is the most correlated with engagement. The framework presented is part of a bigger picture where the generated models are utilised to tailor content generation to a player’s particular needs and playing characteristics.